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Abstract — Internal routing inside an ISP network is tlie foun- 
dation for lots of services that generate revenue from the ISP's 
customers. A fine-grained control of paths taken by network 
traffic once it enters the ISP's network is therefore a crucial 
means to achieve a top-quality offer and, equally important, 
to enforce SLAs. Many widespread network technologies and 
approaches (most notably, MPLS) offer limited (e.g., with RSVP- 
TE), tricky (e.g., with OSPF metrics), or no control on internal 
routing paths. On the other hand, recent advances in the 
research community [1] are a good starting point to address 
this shortcoming, but miss elements that would enable their 
applicability in an ISP's network. 

We extend pathlet routing [1] by introducing a new control 
plane for internal routing that has the following qualities: it 
is designed to operate in the internal network of an ISP; it 
enables fine-grained management of network paths with suitable 
configuration primitives; it is scalable because routing changes 
are only propagated to the network portion that is affected by 
the changes; it supports independent configuration of specific 
network portions without the need to know the configuration 
of the whole network; it is robust thanks to the adoption of 
multipath routing; it supports the enforcement of QoS levels; it is 
independent of the specific data plane used in the ISP's network; 
it can be incrementally deployed and it can nicely coexist with 
other control planes. Besides formally introducing the algorithms 
and messages of our control plane, we propose an experimental 
vaUdation in the simulation framework OMNeT++ that we use 
to assess the effectiveness and scalability of our approach. 

1. Introduction 

It is unquestionable that routing choices inside the network 
of an Internet Service Provider (ISP) are critical for the quality 
of its service offer and, in turn, for its revenue, and several 
technologies have been introduced over time to provide ISPs 
with different levels of control on their internal routing paths. 
These technologies, ranging from approaches as simple as 
assigning costs to network hnks (like, e.g., in OSPF) to real 
traffic engineering solutions (like, e.g., RSVP), usually fall 
short in at least one among: complexity of setup, predictability 
of the effects, and degree of control on the routing paths. 
The research community has worked and still contributes to 
this hot matter from different points of view: control over 
paths is attained by means of source routing techniques; 
besides this, many papers advocate the use of multipath routing 
as a means to ensure resiliency and quick recovery from 
failures; moreover, the granularity of the routing information 
to be disseminated to support multipath and source routing 
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is sometimes controlled by using hierarchical routing mech- 
anisms. However, to the extent of our knowledge, existing 
technological and research solutions still fail in conjugating 
a fine-grained control of network paths, support for multipath, 
differentiation of Quality of Service levels, and the possi- 
bihty to independently configure different network portions, 
a few goals that an ISP is much interested in achieving 
without impacting the simplicity of configuration primitives, 
the scalabihty of the control plane (in terms of consumed 
device memory and of exchanged messages, especially in the 
presence of topological changes), the robustness to faults, and 
the compatibihty with existing deployed routing mechanisms. 

In this paper we propose the design of a new control plane 
for internal routing in an ISP's network which combines all 
these advantages. Our control plane is built on top of pathlet 
routing [1], which we believe to be one of the most convenient 
approaches introduced so far to tackle the ISP's requirements 
described above. 

The foundational principles of our control plane are as 
follows. A pathlet is a path fragment described by a t-uple 
{FID, vi,V2, cr, S), the semantic being the following: a pathlet, 
identified by a value FID, describes the possibility to reach 
a network node V2 starting from another network node vi, 
without specifying any of the intermediate devices that are 
traversed for this purpose. A pathlet need not be an end- 
to-end path, but can represent the availability of a route 
from an intermediate system vi to an intermediate system 
V2 in the ISP's network. An end-to-end path can then be 
constructed by concatenating several pathlets. The S attribute 
carries information about the network destinations (e.g., IP 
prefixes) that can be reached by using that pathlet (given that 
pathlets are not necessarily end-to-end, this attribute can be 
empty). In the control plane we propose, routers are grouped 
into areas: an area is a portion of the ISP's network wherein 
routers exchange all information about the available links, in 
a much similar way to what a link-state routing protocol does; 
however, when announced outside the area, such information 
is summarized in a single pathlet that goes from an entry 
router for the area directly to an exit router, without reveahng 
routing choices performed by routers that are internal to the 
area. This special pathlet, which we call crossing pathlet, is 
considered outside the area as if it were a single hnk. An area 
can enclose other areas, thus forming a hierarchical structure 
with an arbitrary number of levels: the a attribute in a pathlet 
encodes a restriction about the areas where that pathlet is 



supposed to be visible. 

In designing our control plane we took into account several 
aspects, among which: efficient reaction to topological changes 
and administrative configuration changes, meaning that the 
effects of such changes are only propagated to the network 
portion that is affected by them; support for several kinds 
of routing policies; support for multipath and differentiation 
of QoS levels; and compatibility and integration with other 
technologies that are already deployed in the ISP's network, 
to allow an incremental deployment. By introducing areas we 
also offer the possibility for different network administrators to 
independently configure different portions of an ISP's network 
without the need to be aware of the overwhelmingly complex 
setup of the whole network. 

Our contribution consists of several parts. First of all, we 
introduce a model for a network where nodes are grouped in 
a hierarchy of areas. Based on this model, we define the basic 
mechanisms adopted in the creation and dissemination of path- 
lets in the network. We then present a detailed description of 
how network dynamics are handled, including the specification 
of the messages of our control plane and of the algorithms 
executed by a network node upon receiving such messages 
or detecting topological or configuration changes. Further, we 
elaborate on the practical applicability of our control plane in 
an ISP's network in terms of possible deployment technologies 
and propose some possible extensions to accommodate further 
requirements. Last, we present an experimental assessment of 
the scalability of our approach in a simulated scenario. 

The rest of the paper is organized as follows. In Section II 
we review and classify the state of the art on routing mecha- 
nisms that could match the requirements of ISPs. In Section III 
we introduce our formal network model. In Section IV we 
describe the basic pathlet creation and dissemination mecha- 
nisms. In Section V we detail the message types of our control 
plane and describe the network dynamics. In Section VI we 
present applicability considerations and possible extensions to 
accommodate other requirements. In Section VII we present 
the results of our experiments run in the OMNeT++ simulation 
framework. Last, conclusions and plan for future work are 
presented in Section VIII. 

II. Related Work 

Many of the techniques that we adopt in our control 
plane have already been proposed in the literature. Most 
notably, these techniques include source routing (intended 
as the possibility for the sender of a packet to select the 
nodes that the packet should traverse), hierarchical routing 
(intended as a method to hide the details of routing paths 
within certain portions of the network by defining areas), and 
multipath routing (intended as the possibility to compute and 
keep multiple paths between each source-destination pair). 
However, none of the contributions we are aware of combines 
them in a way that provides all the benefits offered by our 
approach. We provide Table I as a reading key to compare 
the state of the art on relevant control plane mechanisms, 
discussed in the following. 
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A CLASSinCATION OF THE STATE OF THE ART ACCORDING TO THE 
ADOPTION OF SOME RELEVANT ROUTING TECHNIQUES. 



Pathlet routing [1] is probably the contribution that is closest 
to our control plane approach: its most evident drawback is 
the lack of a clearly defined mechanism for the dissemination 
of pathlets, which the authors only hint at. Path splicing [2] 
is a mechanism designed with fault tolerance in mind (see 
also [13]): it exploits multipath to ensure connectivity between 
network nodes as long as the network is not partitioned. 
However, actual routing paths are not exposed, and this limits 
the control that the ISP could enforce on internal routing. 
Even in MIRO [11], where multiple paths can be negotiated to 
satisfy the diverse requirements of end users, there can be no 
full control of a whole routing path. NIRA [3] compensates 
this shortcoming, but it is designed only for an interdomain 
routing architecture, hke MIRO, and it reUes on a constrained 
address space allocation, a hardly feasible choice for an ISP 
that is taken also by Landmark [10]. Slick packets [7] is 
also designed for fault tolerant source routing, achieved by 
encoding in the forwarded packets a directed acyclic graph 
of different alternative paths to reach the destination. Besides 
the intrinsic difficulty of this encoding, it inherits the hmits of 
the dissemination mechanisms it reUes on: NIRA or pathlet 
routing. BGP Add-Paths [8] and YAMR [12] also address 
resiliency by announcing multiple paths selected according 
to different criteria, but they only adopt multipath routing, 
provide very limited or no support for hierarchical routing, and 
have some dependencies on the BGP technology. A completely 
different approach is taken by HLP [4], which proposes a 
hybrid routing mechanism based on a combination of link- 
state and path- vector protocols. This paper also presents an in- 
depth discussion of routing policies that can be implemented 
in such a scenario. Although this contribution matches more 
closely our approach, it is not conceived for internal routing 
in an ISP's network, it constrains the way in which areas are 
defined on the network, and it has limits on the configurable 
routing pohcies. A similar hybrid routing mechanism called 
ALVA [9] offers more flexibiUty in the configuration of areas 
but, like Macro-routing [5], it does not explicitly envision 
source routing and multipath routing. HDP [6] is a variant 
of this approach that, although natively supporting Quality of 
Service and traffic engineering objectives, is closely bound 
to MPLS and accommodates source routing and multipath 




Fig. 1. A sample network. Stack labels are integer numbers. Rounded boxes 
represent areas Aa-, with the associated stacks specified as subscript a. 

routing only in the limited extent allowed by this technology. 

Some of the papers we mention here also point out an aspect 
that is key to attain the nice control plane features we are 
looking for: path-vector protocols allow the setup of complex 
information hiding and manipulation policies, whereas link- 
state protocols offer fast convergence with a low overhead. 
Therefore, a suitable combination of the two mechanisms, 
which is considered in our approach, should be pursued to 
inherit the advantages of both. 

III. A Hierarchical Network Model 

We now describe the hiearchical model we use to represent 
the network. We model the physical network topology as a 
graph, with vertices being routers and edges representing links 
between routers: let G = (V, E) be an undirected graph, where 
V is a set of vertices and E = {{u,v)\u,v € V} is a set of 
edges that connect vertices. Fig. 1 shows an example of such 
graph. We assume that any vertex in the graph is interested 
in establishing a path to special vertices that represent routers 
that announce network destinations. Therefore, we introduce 
a set of destination vertices D C V. We highlight that the 
same representation can be adopted to capture the topology of 
overlay networks, while keeping the model unchanged. 

In order to improve scalability and Umit the propagation of 
routing information that is only relevant in certain portions 
of the network, we group vertices into structures called areas. 
To describe the assignment of a vertex u e F to an area we 
associate to z; a stack of labels S{v) = {Iq h ■ ■ ■ In), where 
each label is taken from a set L. To simpUfy notation and 
further reasoning, we assume that lo is the same for every 
S{v) and that _L € L. ± is a special label that we will use to 
identify routing information (actually, pathlets) that represents 
network links. Referring to the example in Fig. 1, we have 
L - {0,1,2,3,±}, S{vi) = S{v2) = S{v3) = (0 1 3), 
S{vi) = S{v5) = (0 1), S{V6) = (0), and S^vr) = (0 2 1). 

We now define some operations on label stacks that allow 
us to introduce the notion of area and will be useful in the 
rest of the paper. Given two stacks ai = {h h ■ ■ ■ h) and 
(T2 = {li+i li+2 ■ ■ ■ In), we define their concatenation as 
c^i o 0-2 = {h h ■■■ h k+i k+2 ■ ■ ■ In)- Assuming that () 
indicates the empty stack, we have that a o () = () o a = ex. 
Given two stacks ui and <T2, we say that <t2 strictly extends 
a I, denoted by (Ti C (T2, if 02 is longer than cti and (T2 starts 
with the same sequence of labels as in a\, namely there exists 



a nonempty stack a such that 02 = o\ o a. We say that (T2 
extends u\, indicated by ui C 0-2, if can be empty. 

We call area A„ a non-empty set of vertices whose stack ex- 
tends a, namely a set A„ such that V?; G : cr E S{v). 
The following property is a consequence of this definition: 

Property 3.1: Given a vertex v G V with stack S{v), v 
belongs to the following set of areas: {^alc E S{v)}. 
Our definition of area has a few interesting consequences. 
First, by Property 3.1, specifying the stack S{v) for a vertex 
V defines all areas A^r such that a C S{v). Thus, areas can 
be convenientiy defined by simply specifying the label stacks 
for all vertices. Considering again the example in Fig. 1, 
the assignment of label stacks to vertices implicitly defines 
areas ^(0), A(o 1), A(o 1 3), -4(o 2), and A(o 2 1) (note that 
^(0 2) = ^(0 2 !))■ Moreover, areas can contain other areas, 
thus forming a hierarchical structure. However, areas can never 
overlap partially, that is, given any two areas Ai and A2, it is 
always Ai C A2 or A2 C Ai. Also, the first label Iq in any 
stack plays a special role, because it is: Ai^i^j = V. 

Areas are introduced to hide the detailed internal topology 
of portions of the network and, therefore, to limit the scope 
of propagation of routing information. As a general rule, 
assuming that the internal topology of an area A„ consists 
of all the vertices in Aa- and the edges of G cormecting those 
vertices, our control plane propagates only a summary of this 
information to vertices outside A^. With this approach in 
mind, we introduce two additional operators on label stacks, 
that are used to determine the correct level of granularity to 
be used in propagating routing information. Given two areas 
Aa^ and Aa^, the first operator, indicated by m, is used to 
determine the most nested area that contains both A„^ and 
A„^, namely the area within which routing information that 
is relevant only for vertices in A^^ and A^^ is supposed 
to be confined: this area is defined by ^<j„n<t6- Referring 
to the example in Fig. 1, the most nested area containing 
both V5 and V7 is As(vs)i^s(v7) — ^(o)- The second operator, 
indicated by is used to determine the least nested area 
that includes all vertices in A^.^ but not those in A^f,, namely 
the area that vertices in A^^ declare to be member of when 
sending routing information to neighboring vertices in A^^'- 
this area is defined by A^^^ai, (in case aa E Ob, such 
an area does not exist and A^^^at, = A^^)- Considering 
again Fig. 1, communicates with V5 as a member of area 
Ag^y.^^^s{v5) = ^(0 2)- We now define the two operators 
formally. Given two arbitrary stacks Oa = {ao ... Oj ... a„) 
and (Tb = (feo • • • hi ... bm) such that oo = &o> ai = 
. . ., Ui = bi for some i < min(rn, n) and a^+i ^ if 
i < min(m,n), we define Ua x cTh = (oq . . . Oj) and 
(7a ^ o"6 = {iq --- o,k) where k = min(i + l,n). We 
extend these definitions in a natural way by assuming that 
M (T6 = fTo N = and ^ = >^ fT6 = (). Be 
aware that m is commutative, whereas ^ is not. 

For each area, a subset of the vertices belonging to the 
area are in charge of summarizing internal routing information 
and propagating it outside the area: these vertices are called 
border vertices. In particular, a vertex u G A^ incident on 



an edge {u, v) such that v ^ Au is called a border vertex 
for area A„. In the example in Fig.l, V2 is a border vertex 
for area ^(o i 3) because v-2, G A(o 1 3), {v2,vq) G E, and 
('6 ^ ^(0 1 3)- Because of Property 3.1, a single vertex can 
be a border vertex for more than one area: in Fig. 1, V2 is 
also a border vertex for area Ai^q 1) because V2 G A(o 1) and 
^ ^(0 1)- Also, by definition it may be the case that a 
neighbor of a border vertex is not a border vertex for any 
areas: looking again at Fig. 1, is not a border vertex. Derived 
from the definition of border vertex, we can state the following 
property: 

Property 3.2: There can be no border vertex for area ^(Zq). 

IV. Basic Mechanisms for the Dissemination of 
Routing Information 

After introducing our network model, we can now illustrate 
how routing information is disseminated over the network. In 
order to do so, we first define the concept of pathlet and 
describe how pathlets are created and propagated. We then 
introduce conditions on label stacks and routing poUcies that 
regulate the propagation of pathlets. 

Pathlets - In order to leam about paths to the various des- 
tinations, vertices in graph G exchange path fragments called 
pathlets [1]. In order to support the definition of areas and 
the consequent information hiding mechanisms, we present an 
enhanced definition of a pathlet that is slightly different from 
the original one. A pathlet tt is a t-uple {FID,vi,V2,<J,S) 
where all fields are assigned by vertex vi : FID is an identifier 
of the pathlet ceiled forwarding identifier, and is unique at vi; 
vi G V is the start vertex; V2 G V\v2 7^ f 1 is the end vertex; 
cr is a stack of labels from L called scope stack, and is a new 
field introduced to restrict the areas where pathlet tt should 
be propagated; and (5 is a (possibly empty) set of network 
destinations (e.g., network prefixes) available at V2. FIDs are 
used to distinguish between different pathlets starting at the 
same vertex vi and are exploited by the data plane of vi to 
determine where traffic is to be forwarded. Even pathlets that 
have the same scope stack and, using different network paths, 
connect the same pair of vertices, can still be distinguished 
based on the FID. We assume FIDs are integer numbers. 

Packet forwarding - Each vertex has to keep forwarding 

state information to support the operation of the data plane. 
Since our control plane has to update these information, we 
now define the forwarding state of a vertex by providing hints 
about the packet forwarding mechanism, which is the same 
presented in [1]. In pathlet routing, each data packet carries in 
a dedicated header a sequence of FIDs: this sequence indicates 
the pathlets that the packet should be routed along to reach the 
destination. When a vertex u receives a packet, it considers 
the first FID in the sequence contained in its header: this 
FID, referenced as / in the following, uniquely identifies a 
pathlet TT that is known at u and that has u as start vertex. 
Now, in the general case pathlet tt may lead to an end vertex 
that is not adjacent to u. Since a pathlet does not contain 
the detailed specification of the routing path to be taken to 



reach the end vertex, before forwarding the packet u has to 
modify the sequence of FIDs contained in the packet header 
to insert such specification: u achieves this by replacing / 
with another sequence of FIDs that indicates the pathlets to 
be used to reach the end vertex of tt. Therefore, the first part 
of the forwarding state of u is a correspondence between each 
value of the FID and a (possibly empty) sequence of FIDs, 
which we indicate as fids^{FID). At this point, u has to pick 
a neighboring vertex to forward the packet to. Since also this 
information is missing in pathlet tt, it must be kept locally 
at vertex u. The second part of the forwarding state of u is 
therefore the specification of the next-hop vertex, namely of 
the vertex that immediately comes after u along tt, which we 
refer to as nhu{FID). Both fidsu and n/t„ are computed by 
the control plane, as explained in the following section. 

Atomic, crossing, and final pathlets - We distinguish 
among three types of pathlets: atomic, crossing, and final. 
A pathlet tt = {FID,Vi,V2,cr,S) is called atomic pathlet if 
its start and end vertices are adjacent on graph G. Atomic 
pathlets carry in the S field the network destinations possibly 
available at V2- They are used to propagate information about 
the network topology and are propagated only inside the most 
nested area that contains both vi and V2. To represent the 
fact that a network link {vi,V2) is bidirectional, two atomic 
pathlets need to be created for that link, one from v\ to V2 
(created by vi) and another from V2 to vi (created by V2). 
Atomic pathlets are always marked by putting the special label 
_L at the end of the scope stack. More formally, an atomic 
pathlet is such that (^1,1^2) G E and 3(t 7^ ()|(t = ct o ±. 
Besides serving as a distinguishing mark for atomic pathlets, 
label _L has been introduced to simplify the description of 
pathlet dissemination mechanisms, because it avoids the need 
to consider several special cases. 

Pathlet TT is a crossing pathlet for area A„ if its start and 
end vertices are border vertices for area A^. Crossing pathlets 
always have 5 = and do not contain label _L in the scope 
stack. A pathlet of this type offers vertices outside A^. (that 
is, whose label stack is strictly extended by a) the possibility 
to traverse Acr without knowing its internal topology: crossing 
pathlets are therefore one of the fundamental building blocks 
of our control plane, as they realize the possibility to hide 
detailed routing information about the interior of an area. 
Since a vertex can be a border vertex for more than one 
area, different pathlets with the same start and end vertices 
can act as crossing pathlets for different areas (they would 
have different scope stacks and FIDs). 

Last, TT is a final pathlet if it leads to some network 
destination available at V2, that is, if (5 7^ 0. Like crossing 
pathlets, final pathlets do not contain label _L in the scope 
stack. Final pathlets are created by a border vertex vi for an 
area A^ to inform vertices outside Aa- about the possibility to 
reach a destination vertex V2 & A^ (1 D. 

Notice that between two neighboring vertices it possible to 

create an atomic, a crossing, and a final pathlet: these pathlets 
are disseminated independently and have each a different role. 



as described above. The type (and, therefore, the role and 
scope of propagation) of these pathlets can be determined 
based on the contents of S and on the presence of the 
special label _L in the scope stack. Since the creation and 
dissemination mechanisms are very similar for crossing and 
final pathlets, in the following we detail only those applied 
to crossing pathlets, assuming that they are the same for final 
pathlets unless differently stated. 

PatMet creation - We now describe how atomic and cross- 
ing pathlets are created at each vertex (similar mechanisms 
are applied for final pathlets). When we say "create" we mean 
that a vertex defines these pathlets, assigns to each of them 
a unique FID, and keeps them in a local data structure, as 
illustrated in Section V. In the following, we also use the 
term "composition" to refer to the creation of crossing and 
final pathlets. 

Each vertex u G V creates atomic pathlets 

{FID,u,v,a o {l.),S) such that {u,v) e E, 
a = S{u) M S{v), and 6 contains the set of network 
destinations possibly available at V2. The scope stack a is 
chosen in such a way to restrict propagation of each atomic 
pathlet up to the most nested area that contains both u and 
V. These pathlets are used to disseminate information about 
the physical network topology and act as building blocks 
for creating crossing and final pathlets. When creating an 
atomic pathlet, vertex u also updates its forwarding state 
with nhuiFID) = v and fids J FID) = (). Looking at the 
example of Fig. 1, creates pathlets {l,V4,vr,,{0 1 -L),0), 
(2,W4,i>2,(0 1 -L),0), and (3, i>4,U6,(0 -L),0) (we assigned 
FIDs randomly). V4 then sets nhy^{l) = 175, nhy^{2) = V2, 
nhy^{3) = ve, and fids^^{l) = fidsy^{2) = fidsy^{3) = (). 

Atomic pathlets can be concatenated to create pathlets 
between non-neighboring vertices. To achieve this, we 
introduce a set chains{Il, u, v, a) that contains all the possible 
concatenations of pathlets taken from a set 11, that start at u 
and end at v, and whose scope stack extends a, regardless 
of FIDs and network destinations. chains{Il,u,v,a) is 
formally defined as the set of all possible sequences of 
pathlets in 11, where each sequence (tti TT2 ... 7r„) is 
finite, cycle-free, and such that tt^ = {FIDi,Wi,Wi^i,<Ti,Si), 
a C (Ti, TTj+i = (F/£)i+i,Wi+i, -u;j+2. 0-^+1,(5^+1), and 
c E CTi+i, with i G {l,...,n — 1}, = u, w^+i = v. 
A border vertex u exploits these concatenations to create 
crossing pathlets, that can be used to traverse the areas that 
u belongs to as if they consisted of a single link. Although u 
may be a border vertex for several areas, it creates crossing 
pathlets only for those areas that u's neighbors are actually 
interested in traversing. To find out which are these areas, 
we must consider how u appears to its neighbors: we assume 
that each neighbor n of u that is not in Ag^u) considers u 
as a member of the least nested area that includes u but 
not n, that is, area As[u)^S(n)- For this reason, u creates 
a set of crossing pathlets for each area A = As(u)^s{n)- 
these pathlets start at 11 and end at any other border vertex 
V for A, V u. Similarly, u creates final pathlets that start 



at u and end at any other destination vertex v E D (1 A. In 
the example in Fig. 1, vq considers V2 as a member of area 
A(o 1 3)^(o)=(o 1), whereas V4 considers V2 as a member of 
area A(o 1 3)^(0 i)=(o 1 3)- For this reason, V2 will create 
crossing and final pathlets for 1) to be offered to vq and 
crossing and final pathlets for A(o i 3) to be offered to V4. 
More formally, for each neighbor n, a border vertex u & Acr 
creates crossing pathlets by populating a set crossing ^{11, a), 
with a = S{u) ^ S{n). Each set crossing ^{H, a) contains 
a pathlet tt = {FID,u,w,a,S) for each border vertex 
u; ^ w for A„ and for each sequence (tti 772 ... 7r„) in 
set chains {TLjUjW, a). FID is chosen in such a way to 
be unique at u and 6 is set to the empty set 0. Assuming 
that TTi = {FIDi,Ui,Vi,ai,Si), the forwarding state of u is 
updated by setting fids^{FID) = {FID2 FID3 ... FID^) 
and nhu{FID) = nhu{FID{) = Vi. Note that, in general, 
pathlet TTi may not be an atomic pathlet: in this case, u 
has to recursively expand tti into the component atomic 
pathlets in order to get the correct sequence of FIDs to be 
put in fids^{FID) and the correct next-hop to be assigned 
as nhu{FID). However, because of the way in which set 
chains {li,u^w, a) will be used in the following, and in 
particular because of the composition of set 11 on which it 
will be constructed, we assume without loss of generality that 
TTi is always an atomic pathlet. As an example taken from 
Fig. 1, let n = {(2, V2, Vi, (0 1 ±), 0) , (3, Vi, ^5, (0 1 ±), 0) , 
{l,vuV3,{0 1 3 ±),0), (2,t>3,?;5,(0 1 ±),0)}. V2 may have 
in its set crossing 1)) a pathlet (1,^2, ^5,(0 1),0) 
corresponding to the sequence of atomic pathlets 
((2,w2,f4,(0 1 -L),0) (3,z;4,^;5,(0 1 -L),0)) taken from set 
chains {\i,V2,V5, (0 1)). V2 will therefore set fidSy^{l) = (3) 
and n/it,2 (1) = V4. 

Final pathlets are created in a much similar way as crossing 
pathlets, except that they are composed towards vertices in 
A^ (1 D and 6 is set to the set (5„ of network destinations of 
the last component pathlet in the sequence. Final pathlets are 
put in a set final^{Il, a). 

Because of the way in which pathlets are created and of the 
fact that there are no crossing or final pathlets for area 
(Property 3.2), we can easily conclude that there are always 
at least two labels in the scope stack of any pathlet. This is 
stated by the following property: 

Property 4.1: For any pathlet {FID,u,v,a,6) there exists 
d- ^ {) such that a = (Zq) ° ^■ 

Discovery of border vertices - In order to be able to 
compose crossing pathlets for an area, a border vertex u must 
be able to discover which are the other border vertices for 
the same area. The only information that u can exploit to this 
purpose are the pathlets it has received. Given that a border 
vertex connects the inner part of an area with vertices outside 
that area, a simple technique to detect whether a vertex w is a 
border vertex consists therefore in comparing the scope stacks 
of suitable pairs of pathlets that have u as a common vertex. 

The technique is based on the following lemma. 

Lemma 4.1: If a vertex u & A^ receives two 



Algorithm 1 Algorithm that a vertex u E Ac can use to 
discover remote border vertices for based on the known 
pathlets in E. 

1: function DiscoverBorderVertices(u, a, 11) 

2: B ^ 

3: for each pair (7ri,7r2) of pathlets with tti = 
(FIDi,i;i,ii;i,cri,5i) and -ki = {FID2,V2, 1^2,(12,32), 
such that vi ^ V2 or Wi ^ W2> and 3v such that both 
TTi and 7r2 start or end at v do 

4: if 3ai 7^ such that ai = ai o (/), / g L, and 

3ct2 9^ such that 0-2 = CT2o(_L) and a\= a and (T2 C cr 
then 

5: B ^ B U 

6: end if 

7: end for 
8: return B 
9: end function 



pathlets TTi = (f/I'i,t;i,ti;i,(Ti o and 
7r2 = {FID 2, V2,W2, (72 {!.),%), with Z e L, cti 7^ (), 
(72 7^ (), the start and end vertices of tti and 112 are such that 
vi ^ V2 or wi ^11)2, the scope stacks of tti and 7r2 are such 
that (72 C cr = CTi, and there exists a vertex w such that both 
TTi and 7r2 start or end at v, then u is a border vertex for A^. 

Proof: The statement follows from the way in which 
scope stacks are assigned to pathlets. The fact that v G 
{vi,Wi} implies that v G A„^: in fact, if / = _L, then tti 
is an atomic pathlet whose scope stack is therefore assigned 
in such a way that ui = S{vx) m S{wi)\ since we know that 
S{vi) M S{wi) C S{v), by Property 3.1 we can conclude 
that V G Ao-j . Otherwise, if Z 7^ _L, then tti is either a crossing 
pathlet for some area A^^oii) or a final pathlet; in both cases, 
being an endpoint of pathlet tti, v must belong to Ac^o[i) and, 
using Property 3.1 again, this also impUes that v e A^-^^. Since 
(j\ = cr, we can conclude that v G A^. On the other hand, 
from the scope stack (T2 C u of the atomic pathlet tt2 we 
know that v has some neighbor that is not in A^: this makes 
V a border vertex for A^. ■ 
According to this lemma, a vertex u E A^ can use the fol- 
lowing simple algorithm, formalized as function DlSCOVER- 
BorderVertices(w, fj, 11) in Algorithm 1, to discover other 
border vertices for A^ based on a set of known pathlets 11: 
consider any possible pair (tti, ^2) of pathlets in 11 whose start 
and end vertices have exactly one vertex v in common; if this 
pair satisfies the conditions of the lemma, w is a border vertex 
for A„. 

Routing policies - So far we have described how to 

compose crossing and final pathlets by considering all the 
possible concatenations of available pathlets. Although this 
produces the highest possible number of altemative paths, 
resulting in the best level of robustness and in the availability 
of different levels of Quality of Service, depending on the 
topology and on the assignment of areas it can be demanding 
in terms of messages exchanged on the network and of pathlets 
kept at each router. However, our control plane can also easily 



accommodate routing policies that influence the way in which 
pathlets are composed and disseminated. We stress that these 
poUcies can be implemented independently for each area: that 
is, the configuration of routing policies on the internal vertices 
of an area may have no impact on the routing information 
propagated outside that area. We believe this is a significant 
relief for network administrators, who do not necessarily need 
any longer to keep a complete knowledge of the network setup 
and to perform a complex planning of configuration changes. 

We envision two kinds of policies: filters and pathlet com- 
position rules. Filters can be used to restrict the propagation of 
pathlets. For example, the specification of a filter on a vertex u 
can consist of a neighboring vertex v and a triple {wi,W2, cr): 
when such a filter is appUed, u wiU avoid propagating to v all 
those pathlets whose start vertex, end vertex, and scope stack 
match the triple. 

Pathlet composition rules can be used to affect the creation 
of crossing and final pathlets. We describe here a few possible 
pathlet composition rules. As opposed to the strategy of 
considering all the possible concatenations of pathlets, a border 
vertex v can create, for each end vertex w of interest, only 
one crossing (or final) pathlet that corresponds to an optimal 
sequence of pathlets to that end vertex. Several optimality 
criteria can be pursued. For example, v could select the 
shortest sequence of pathlets by running Dijkstra's algorithm 
on the graph resulting by the union of the pathlets it knows. 
We highlight that, with this approach, v can still keep track 
of possible alternative paths but does not propagate them as 
pathlets: in case the shortest sequence of pathlets to a certain 
vertex w is no longer available (for example because of a 
failure), v can transparently switch to an altemative sequence 
of pathlets leading to w by just updating the forwarding state 
and without sending any messages outside its area Agf^^y 
As a variant of this approach, pathlets can be weighted 
according to performace indicators (delay, packet loss, jitter) 
of the network portion they traverse: in this case the optimal 
sequence of pathlets corresponds to the one offering the best 
performance. Alternatively, pathlets can be weighted according 
to their nature of atomic or crossing pathlet: assuming that 
atomic pathlets are assigned weight and crossing pathlets 
are assigned weight 1, the optimal pathlet tries to avoid 
transit through areas. Another pathlet composition rule could 
accommodate the requirement of an administrator that wants to 
prevent traffic from a specific set V of vertices from traversing 
a specific area A. Since detailed routing information about the 
interior of an area is not propagated outside that area, it may 
not be possible to estabUsh whether a specific pathlet traverses 
A or not. Therefore, to implement this pathlet composition 
rule, pathlets could carry an additional attribute that is a set of 
shaded vertices: crossing pathlets for A disseminated by the 
border vertices of A will have the set of shaded vertices set 
to V; upon receiving a pathlet, a vertex v will check whether 
v's identifier appers in the set of shaded vertices and, if so, 
will refrain from using that pathlet for composition or for 
sending traffic. A similar mechanism could be implemented 
to prevent traffic to specific destinations from traversing A: 



in this case, a set of shaded destinations could be carried 
in the pathlets instead. Of course, the two techniques can be 
combined by using both the set of shaded vertices and the set 
of shaded destinations: in this way, a set of vertices V can be 
prevented from traversing an area A to send traffic to specific 
destinations. 

Pathlet dissemination - All the created pathlets are dis- 
seminated to other vertices in G based on their scope stacks, 
as explained in the following. Consider any pathlet tt = 
{FID,u,v,a,6) and let cr = ct o (I) (by Property 4.1, such 
a ^ and / € L must exist). The dissemination of tt is 
regulated by the following propagation conditions. A vertex 
w can propagate tt to a neighboring vertex n either if n = u 
or if tt's scope stack does not satisfy any of the following 
conditions: 

1) S{w) N S{n) C a: restricts propagation of any pathlets 
outside the area in which they have been created; 

2) cr C S{w) M S{n): prevents propagation of crossing and 
final pathlets inside the area of the vertex that created 
them; 

3) a = S{n) ^ S{w): prevents w ^ A from propagating 
crossing and final pathlets for A inside A; 

4) n = v: prevents sending to n a pathlet that is useless 
for n. 

Conditions 2), 3), and 4) are introduced to prevent the propa- 
gation of pathlets to vertices that would never use them, thus 
limiting the amount of exchanged information during pathlet 
dissemination. Condition 1) can be expressed from the point 
of view of a single vertex, leading to the following invariant: 

Property 4.2: All the pathlets received by a vertex v have 
a scope stack a' = a o (I) such that a C S{v). 

For convenience, given a vertex w that is assigned scope 
stack S{w) = (Tiu, we define N{w, a^,, a) as the set of 
neighbors of w to which w can propagate a pathlet with scope 
stack a according to the propagation conditions and to the 
routing policies. We assume that N{w,ayj,{)) = for any 
a^. 

So far we have mentioned that the propagation conditions 
regulate the propagation of pathlets. However, we will see 
in Section V that other kinds of messages exchanged by 
our control plane are also propagated according to the same 
conditions. 

Example of pathlet creation and dissemination - To 

show a complete example of creation and dissemination of 
pathlets, consider again the example in Fig. 1 and let cn host 
network destination d. In the following we assume that there 
are no filters applied, that the pathlet composition rule is to 
compose all possible sequences of pathlets (although we show 
only some of them), and that FIDs are randomly assigned 
integer numbers, yet obeying the rules specified in this section. 
The atomic pathlet 1124, ± — {1,V2,V4, (0 1 -L),0), created by 
vertex V2, is propagated by V2 to V3 because S{v2) x S{v3) = 
(0 13)^ (0 1), (0 1 _L) g (0 1 3), (0 1 _L) ^ 5(^3) ^ 
S{v2) = (0 1 3), and W3 7^ V4, it is also propagated to vi 
for the same reasons. Instead, 1:24,1. is not propagated by 



V2 to vq because S{v2) n S{vq) = (0) C (0 1) (the first 
propagation condition applies), and it is not propagated by V2 
to V4 because the end vertex of 7r24,_L is V4 itself. For similar 
reasons, 7724, _l is further propagated by to W5, but in turn 

does not propagate it to v^. Therefore, the visibility of 
7r24,_L is restricted to vertices inside yl(o 1). In a similar way, 

creates the atomic pathlets TTsa^j, = {2,v^,vz,{Q 1 -L),0) 
and 7r54__L = (3, W5, 1)4, (0 1 _L), 0), while V4 creates the 
atomic pathlet 7r46,_L = (3, 1^4, ve, (0 _L), {d}). The reader 
can easily find how these atomic pathlets are propagated. 
As a border vertex of 1 3)> ^^3 will also propagate 
to W5 the crossing pathlet -K32 = (1, "Ws, ^'2. (0 1 3), 0) for 
area Ag(^y^)^g^y^)^(Q i 3). Once pathlets have been dissem- 
inated, has learned about a set of pathlets 11 and can 
create a crossing pathlet for area As(vz)^s(v7)={o 1) that 
can be offered to wy. For example, V5 can pick sequence 
(t^ss.j. 7r32 7r24,_L) from chains{\i,v^,v4,S{v^) ^ S{vr)) 
and create in its set crossing {II, S{v 5) ^ S{v7)) the 
crossing pathlet 1154 ~ (1, U5, W4, (0 1), 0). Propagation of this 
pathlet by to V4 is forbidden by the second propagation 
condition, because (0 1) C S{v5) m S{v4) = (0 1), and 
also by the fourth propagation condition, because V4 is also 
the end vertex of 11^4; 1^54 will however be propagated by 
to vr because Sivr,) n S{v7) ^ (0) (0), (0 1) g (0), 
(0 1) ^ S{vr) ^ S{V5) = (0 2), and ^ W4. To provide 
an alternative path, ^5 can create another crossing pathlet 
""54 = (9) ■'^5) ^^4; (0 1), 0), corresponding to the sequence 
consisting of the single atomic pathlet (7r54 and propagated 
in the same way as 7r54. Last, wy will also create an atomic 
pathlet 7r75^_L = (8,W7,W5,(0 -L),0). At this point, vr has 
two ways to construct a path from itself to vertex vq, which 
contains destination d: it can concatenate pathlets tt75,±, -K^i, 
and 7746, _L or pathlets 7775 j^, 77^4, and 7745 j^. The availability of 
multiple choices supports quick recovery in case of fault and 
allows 117 to select the pathlet providing the most appropriate 
Quahty of Service. 

V. A Control Plane for Pathlet Routing: 
Messages and Algorithms 

We now describe how the dissemination mechanisms il- 
lustrated in Section IV are reaUzed in terms of messages 
exchanged among vertices and algorithms executed to update 
routing information. In this section we also detail how to 
handle network dynamics, including how to deal with topolog- 
ical changes and administrative reconfigurations. This actually 
completes the specification of a control plane for pathlet 
routing. 

A. Message Types 

First of all, we detail all the messages that are used by 
vertices to disseminate routing information. Each message 
carries one or more of the following fields: s: a stack of 
labels; d: a set of network destinations; p: a pathlet; f: a 
FID; a: a boolean flag (which tells whether a vertex has "just 
been activated"). We assume that every message includes an 
origin field o that specifies the vertex that first originated the 



message. Messages can be of the following types, with their 
fields specified in square brackets: 

• Hello [s, d, a] - Used for neighbor greetings. It carries 
the label stack s of the sender vertex, the set of network 
destinations d originated by the sender vertex, and a flag 
a which is set to true when this is the first message sent 
by a vertex since its activation (power-on or reboot). Un- 
like other message types. Hello is only sent to neighbors 
and is never forwarded. Moreover, in order to be able 
to detect topological variations, it is sent periodically by 
each vertex. 

• Pathlet [p] - Used to disseminate a pathlet p. 

• Withdrawlet [f , s] - Used to withdraw the availability 

of a pathlet with FID f , scope stack s, and start vertex o. 
We assume that this message can only be originated by 
the vertex that had previously created and disseminated 
the pathlet. 

• Withdraw [s] - Used to withdraw the availability of all 
pathlets having s as scope stack and o as start vertex. 

In order to keep disseminated information consistent in the 
presence of faults and reconfigurations, we assume for con- 
venience that all vertices in the network have a synchronized 
clock, and we call T its value at any time. Every message type 
but Hello has a timestamp field t that, unless otherwise stated, 
is set by the sender to the current clock T when sending a 
newly created message; the timestamp is left unchanged when 
a message is just forwarded from a vertex to another. The 
purpose of the timestamp is to let vertices discard outdated 
messages, which is especially important in the presence of 
faults. In practice, a local counter at each vertex can be used 
in place of the clock value, and its value can be handled in 
a way similar to OSPF sequence numbers (see in particular 
Section 12.1.6 of [14]). 

With the exception of Hello, messages also have a source 
field src containing the identifier of the vertex that has sent 
(or forwarded) the message. This field is also used to avoid 
sending the message back to the vertex from which it has 
been received (a technique similar to the split horizon adopted 
in commercial routers). Since the Hello message is never 
forwarded by any vertices, it contains only the origin field. 

Given their particular nature, in the following we omit 
specifying for each message how the origin, timestamp, and 
source fields are set, unless we need exceptions to their usual 
assignment. 

B. Routing information stored at each vertex 

In our control plane, no vertex has a complete view of all the 
available routing paths. However, as a partial representation 
of the current network status, each vertex u ^ V keeps the 
following information locally: 

• For each neighbor v £ V such that (7/, v) E E, a label 
stack Su{v) that u currently considers associated with v 
and a set Dy_{v) of network destinations originated by v. 

• A set n„ of known pathlets, consisting of the atomic 
pathlets created by u and of pathlets that u has received 



from neighboring vertices, u can concatenate these path- 
lets to reach network destinations and, in case it is a 
border vertex, to compose and disseminate crossing and 
final pathlets. Each pathlet tt G n„ is associated an expiry 
timer Tp(7r), that specifies how long the pathlet is to be 
kept in n„ before being removed. When a new pathlet is 
created by u, its expiry timer is set to the special value 
Tp(7r) = 0, meaning that the pathlet never expires. 

• For every area A„ for which u is a border vertex, a set 
-Bu(o") of vertices v G Aa-, v 7^ u, that are also border 
vertices for A^^, and sets C„(cr) and Fu{a) that contain, 
respectively, the crossing and final pathlets for area A„ 
composed by u. 

• A set Hu, called history, that tracks the most recent 
piece of information known by u about each pathlet 
(i.e., not just pathlets in n„). This set consists of t- 
uples {FID, V, a, t, type), where: the FID and the start 
vertex v identify a pathlet tt with scope stack a; t is the 
timestamp of the most recent information that u knows 
about TT (it may be the time instant of when tt has been 
composed or deleted by u, or the timestamp contained 
in the most recent message received by u about tt); 
and type e {+, — } determines whether the last known 
information about tt is positive (tt has been composed by 
u or a Pathlet message has been received about tt) or 
negative (n has been deleted by u or a Withdrawlet or 
Withdraw message has been received about tt). 

The reason why we have introduced an expiry timer Tp(7r) 
for each pathlet tt in n„ is that we want to prevent indefinite 
growth of n„. In fact, there may be pathlets that can no longer 
be used by u for concatenations and for which u may never 
receive a Withdrawlet or Withdraw message: the expiry 
timer is used to automatically purge such pathlets from n„. 
This situation can occur when a vertex or a link is removed 
from G. For example, consider the network in Fig. 1 and 
suppose that V2 composes and announces a crossing pathlet 
7r25 = (7, V2, V5, (0 1), 0) for area >l(o 1). If link {v2, vq) fails, 
vq has no way to receive a Withdrawlet for 7725, because only 
V2 can originate this message and the propagation conditions 
prevent it from being forwarded inside area A(o 1). However, 
vq can no longer use 7725 for any concatenations and therefore 
has no reason to keep this pathlet in its set n„g : 7125 can indeed 
be automatically removed after timer Tp{n25) has expired. 
The configuration of our control plane therefore requires the 
specification of a timeout value called pathlet timeout: this 
is the value to which the expiry timer Tp(7r) of a pathlet 
TT is initialized when Tp{iT) is activated (we will see in the 
following when this activation occurs). 

Also the history i?„ could grow indefinitely, because an 
entry is stored and kept in Hu even for each deleted or 
withdrawn pathlet. Therefore, our control plane also requires 
the specification of a history timeout: this value determines 
how long negative entries (i.e., with type = — ) in the history 
Hu of any vertex u are kept before being automatically purged 
from Hu. Positive entries (with type = +), on the other hand. 



never expire. 

In principle, we could completely avoid timeouts and 
remove pathlets and history entries immediately. However, 
this would significantly increase the number of exchanged 
messages and cause the deletion of pathlets that should instead 
be preserved, even in normal operational conditions. Consider 
again Fig. 1 and assume there is no pathlet expiry timer. If 
^6 received only pathlet ttst^j, = {Q,V5,vy,{0 -L),0) before 
receiving the crossing pathlet 7r25, vq would inunediately 
withdraw ttsy^j, because it cannot use it for concatenations 
and it may never receive a Withdrawlet or Withdraw for that 
pathlet. A similar argument applies to the history timer. Look 
back at Fig. 1 and assume there is no history expiry timer. Note 
that with this assumption negative entries are just not kept in 
the history, actually defeating its purpose. Suppose that, after 
disseminating an atomic pathlet nQ2.± — {l,ve,V2, {0 -L),0) 
to the whole network, vertex ug withdraws this pathlet using a 
Withdrawlet message (for example because link {v2,vq) has 
failed). Also suppose that rebooted before being able to 
forward the Withdrawlet to v-j: pathlet nQ2,± would thus be 
held in set . When becomes again active, it receives a 
Pathlet message containing nfy2.± from vj, and has no way to 
determine that such information is out-of-date. The only way 
is to propagate pathlet nQ2,± to all applicable vertices until it 
reaches wg, which can again withdraw it from the network. 

For the sake of clarity, we specify here the strategy with 
which the history is updated when creating or deleting pathlets, 
and avoid mentioning it again, unless there are exceptions to 
this strategy. Every time a pathlet tt = {FID,u,v,a,5) is 
created by a vertex u, the history of u is automatically 
updated with a positive entry {FID,u,<t,T,+), where T 
denotes the time instant of the creation. If an entry for the same 
FID and start vertex u already existed in Hu, that entry is 
replaced by this updated version. When pathlet tt is no longer 
available (for example because u has detected that some of 
the component pathlets are no longer usable), is updated 
with a negative entry {FID, u,a,T,—), where T denotes the 
time instant in which tt has become unavailable. This entry 
replaces any previously existing entry referring to the same 
pathlet TT. We recall that this negative entry is automatically 
removed from Hu after the history timeout expires. 

C. Algorithms to Support Handling of Network Dynamics 

Before actually describing how network dynamics are han- 
dled, we introduce a few algorithms that vertices execute when 
they detect a change of the locally maintained routing informa- 
tion. In particular, we describe the operations performed by a 
vertex u when its sets n„, C„, or F„ are updated. Most of the 
events that may trigger such updates, including topological 
changes and administrative reconfigurations, can be handled 
based on the algorithms described in this subsection. We 
discuss in detail the application of these algorithms to handle 
specific events in the following subsections. 

Suppose the set n„ of currently known pathlets at a vertex 
u is changed and is to be replaced by a new set of known 
pathlets Unew u then undertakes the following actions, for- 



malized as procedure UpdateKnownPathlets(w, Unew) in 
Algorithm 2: u sends messages to its neighbors to disseminate 
pathlets that are newly appeared in Unew (with respect to n„) 
and withdraw those that are no longer in this set. Note that, 
while withdrawn pathlets are immediately removed from n„ 
at the end of the algorithm, the corresponding forwarding 
state is only cleared by u after a timeout T/, in order to 
allow correct forwarding of data packets while the withdraw is 
propagated on the network (note that the statement at line 28 of 
Algorithm 2 is non-blocking). Of course some packets could 
be lost if the withdrawn pathlets are physically unavailable. 
Pathlets that existed in n„ but have their scope stack or set 
of destinations updated in Unew are handled by u in a special 
way: for each of these pathlets u disseminates the updated 
instance of the pathlet to selected neighbors (according to 
the propagation conditions and to the rouing policies set at 
u), and withdraws the old instance of the pathlet from other 
neighbors to which the new instance of the pathlet cannot be 
disseminated. After having updated Hu with the contents of 
n„ew. u checks whether the start vertex of each pathlet in n„ 
is still reachable: it does so by concatenating arbitrary pathlets 
in n„, regardless of their scope stacks. If the start vertex of 
some pathlet tt is found to be unreachable, or if including tt in 
any sequences of pathlets would always result in a cycle, u can 
no longer use pathlet tt for composition or traffic forwarding, 
and it schedules automated deletion of the pathlet from n„ by 
initizializing its expiry timer Tp(7r). For all the other pathlets, 
the expiry timer is reset, meaning that they will never expire. 
Last, u checks whether the component pathlets of its crossing 
and final pathlets are still available and whether new crossing 
or final pathlets can be composed, and updates sets C„ and 
Fu accordingly. The latter step requires further actions, which 
are detailed in the following procedure. 

When set n„ is replaced by Unew, vertex u must also 
check whether the component pathlets for its crossing and 
final pathlets are still available in Unew and whether there 
are new crossing and final pathlets that u should compose 
due to newly appered pathlets in Unew- Function IsPath- 
letComposable(w, tt, n, E) in Algorithm 3 can be used 
to establish whether a certain pathlet tt can (still) be com- 
posed by u based on the set n of known pathlets at u 
and on a set E of admissible end vertices for tt (the check 
performed by this function actually reflects the mechanism 
for the construction of set crossinQu as explained in Sec- 
tion IV). The composition (or deletion) of crossing and final 
pathlets also depends on the areas for which tt is a border 
vertex and on the knowledge of other border vertices. All 
the operations that u is supposed to execute to update its 
crossing and final pathlets are therefore formalized as proce- 
dure UpdateComposedPathlets(w, U-new) in Algorithm 4: 
u considers pathlets that it can no longer compose (Coid) 
because it is no longer a border vertex for some area or because 
some of the component pathlets are no longer available in 
Unew'- u- niay also compose new crossing and final pathlets 
(Cnew) because it has become a border vertex for some area 
or because there are new possible compositions of pathlets 



Algorithm 2 Algorithm to update the set !!„ of known pathlets at a vertex u. The first procedure addresses the case when the 
label stack of u is contextually changed from Som to Snew, whereas the second only reaUzes the update of n„. 



1: procedure UpdateKnownPathletsAndStack(«, Som, Snew, n„e™) 

2: for each n = {FID, v, w, a, 5) G Ilnew\^u do 

3: We are considering a pathlet n that is not in Il„ but is in Il„ew (new pathlet) or the updated instance of a pathlet that is both 

in n„ and in Ylnew 

4: it u = V then 

5: Update u's forwarding state according to the composition of 7r 

6: M <r- new Pathlet message 

7: M.p TT 

8: for each n e N{u, Snew, f) do 

9: Send M to neighbor n 

10: end for 

11: end if 

12: end for 

13: for each ttow = {FID, v, w, (Joid,5oid) G Ylu\^new do 

14: it u = V then 

15: M new Withdrawlet message 

16: M.f FID 

17: M.s-*-cToW 

18: if 37r„ew = {FID, v, w, anew, Snew) e n„em then 

19: We are considering a pathlet ixou that is in n„ and has an updated instance 'Knew in Unew 

20: for each n G A^(w, S ) do 

21: Send M to neighbor n 

22: end for 

23: else 

24: We are considering a pathlet TVoid that is in Wu but has been removed in Wnew 

25: for each n G N{u, Sold, (Tou) do 

26: Send M to neighbor n 

27: end for 

28: Clear fids^{FID) and nhu{FID) after a timeout T/ 

29: end if 

30: end if 

31: end for 

32: n„ ^ n 

new 

33: for each tt = {FID, v, w, a, S) G n„ do 

34: if chains{Tlu, u, v, ()) = or any concatenation of a pathlet in chains{H„, u, v, ()) with pathlet tt has a cycle then 

35: 'rp{n) <— value of the pathlet timeout parameter 

36: else 

37: Tp(7r) 

38: end if 

39: end for 

40: UPDATECOMPOSEDPATHLETSANDSTACKCm, Sold, Snew, Hu) 

41: end procedure 

42: procedure UpdateKnownPathlets(u, n„e„) 

43: UPDATEKNOWNPATHLETSANDSTACK(w, S{u), S{u), Unew) 

44: end procedure 



Algorithm 3 Algorithm to check whether a pathlet tt can (still) 
be composed by a vertex u given a set 11 of known pathlets 

and a set E of admissible end vertices for tt. 

1: function IsPathletComposable(u, tt, H, E) 
2: Let tt = {FID, u, v, tr, 6) 

3: if V £ E and 3{iri 1T2 ■ ■ ■ 7r„) € chains {Il,u,v, a) such 
that TTi = {FIDi, Ui, Vi, Gi, 6i), i = 1, . . . ,n and fids^{FID) = 
{FID2 FID3 . . . FIDn) and nhu{FID) = U2 and the pathlet 
composition rules allow composition of 7r then 

4: return True 

5: else 

6: return False 

7: end if 
8: end function 



in T\-new If possible, u attempts to transparently replace 
pathlets in Cou with newly composed pathlets from Cnew 
by just updating its forwarding state and without sending 
any messages; if this is not possible, u withdraws the no 
longer available pathlets from those neighbors to which they 
had been disseminated and clears the forwarding state for 
these pathlets after a timeout Tj (the statement at line 41 of 
Algorithm 4 is non-blocking). Last, u disseminates to selected 
neighbors (according to the propagation conditions and the 
routing policies) all those newly composed pathlets in Cnew 
that were not used as a replacement for pathlets in Cqu- 
Pathlet composition at line 12 of Algorithm 4 follows the 
same mechanism as for set crossing: we did not use set 
crossing{Y\new,(y) here because the FIDs of already existing 



Algorithm 4 Algorithm to update the sets of crossing and final pathlets composed by a vertex ?/. The first procedure considers 
the case when the label stack of u is contextually changed from Som to Snew, whereas the second only reaUzes the update of 
crossing and final pathlets. 



1: procedure UpdateComposedPathletsAndStack(m, Sou, Snew, ^new) 
2: for each area do 

3: Cnew ^ 

4: Cold ^ 

5: if M is a border vertex for then 

6: By,{cr) <- DISCOVERBORDERVERTICES(w, ct, Il„ew) 

7: if C„((t) =0 tlien 

8: Vertex u has become a border vertex for A^ or has not yet composed any pathlets for that area; pathlets in set 

crossing ^{Ilnew,o-) below have end vertices in Bu{cr) 

9: C new <r- crossing ^{'n.new,cr) 

10: else 

11: Vertex u continues to be a border vertex for A„, but it has to refresh available crossing pathlets according to the contents 

of ^^new 

12: Cnew new crossing pathlets not in C„(cr), that u can compose towards vertices in Bu{<t) using pathlets in Il„ew and 

according to the pathlet composition rules 
13: Cold <- {ttItt € Cu{cr) and not IsPathletComposable(w, 7r,n„era,S„(o-))} 

14: end if 

15: else if C„ (a) 7^ then 

16: Vertex u was a border vertex for A^ but is no longer 

17: Cold ^ C„(a) 

18: end if 

19: Update u's forwarding state for any pathlet in Cnew 

20: if C„M = C„(cr) then 

21: All the crossing pathlets have been removed: this piece of information can be propagated with a single Withdraw message 

22: M a new Withdraw message 

23: M.s-*-cT 

24: for each n € N{u, Sold, o") do 

25: Send M to neighbor n 

26: end for 

27: else 

28: for each ttom = {FIDoid,v, w, a, 6) € Coid do 

29: if 3 (FID 

new ; V, w, a, 5) € Cnew\Coid then 

30: Use an alternative pathlet -Knew to transparently replace a no longer available pathlet Woid by only updating u's 

forwarding state 

31: fids^{FIDoid) ^ fids^iFIDnew) 

32: nhu{FIDoid) <r- nhu{FIDnew) 

33: Cnew •<— {Cnew\{T^new}) U {iXold} 

34: else 

35: M ^ a new Withdrawlet message 

36: M.f ^ FID old 

37: M.s <r- a 

38: for each n e I>!(u, Sold, cr) do 

39: Send M to neighbor n 

40: end for 

41: Clear fids^{FIDoid) and nhu{FID oid) after a timeout T/ 

42: end if 

43: end for 

44: for each -Knew = {FID nc,,,, v., w, a, 5) € C„ew\Coid do 

45: M <— a new Pathlet message 

46: Af .p ^ TT 

47: for each n £ N{u, Snew,cr) do 

48: Send M to neighbor n 

49: end for 

50: end for 

51: end if 

52: Cn{(j) ^ {Cu{o)\Cold) U Cnew 

53: end for 

54: Repeat the same steps replacing set Cu{<y) with Fu{a), set Bu{a) with Aa Pi D, set crossing (n„ew,o-) with final(nnew,o^), and 

"crossing pathlets" with "final pathlets" 
55: end procedure 

56: procedure UpdateComposedPathlets(u, Unew) 

57: UPDATEC0MP0SEDPATHLETSANDSTACK(«, S{u), S{u), Unew) 

58: end procedure 



pathlets in C„(cr) must be retained. 

D. Handling Topological Variations and Configuration 
Changes 

As soon as a vertex u becomes active on the network, it 
sends a Hello message M to all its neighbors, with M. s set to 
its label stack S{u), M.d set to the available destinations at u 
(if any), and M.a = True. With this simple neighbor greeting 
mechanism, each vertex can learn about its neighborhood. 
Once a vertex has collected this information, it starts creating 
and disseminating atomic pathlets as explained in Section IV. 
Although this reasonably summarizes the behavior of a vertex 
that has just appeared on the network, "becoming active" is 
just one of the possible topological variations that graph G 
may undergo during network operation. Moreover, our control 
plane must also support administrative configuration changes 
that can occur while the network is rurming. 

In our model, most topological variations and configuration 
changes can be represented as a change of label stacks, in the 
following way: addition of a hnk {u,v) is modeled by the 
assigrmient of value S{v) to label stack Su{v) and of value 
S{u) to label stack Sv{u); removal of a link (m, v) is modeled 
as a change of label stacks Suiv) and Sy{u) to the empty 
stack (); addition and removal of a vertex are modeled as 
a simultaneous addition or removal of all its incident edges; 
an administrative configuration change that modifies the label 
stack S{v) assigned to a vertex v is modeled as an update of 
stacks Sw{v) of all the neighbors w of v. To complete the 
picture of possible reconfigurations, we assume that a change 
in the routing pohcies of a vertex causes a reboot of that vertex 
(this assumption can be removed, but then each vertex has to 
keep track of the pathlets it has propagated): we therefore 
do not discuss this kind of configuration change further. For 
these reasons, we can handle all relevant network dynamics 
by defining a generic algorithm to deal with a change of 
the known label stack of a vertex. We will see in the rest 
of this section that this algorithm is designed to limit the 
propagation of the effect of a network change: in fact, only 
those pathlets that involve vertices affected by the change are 
disseminated as a consequence of the change. Moreover, we 
enforce mechanisms to transparently replace a pathlet that 
is no longer available without the need to disseminate any 
information to the rest of the network. 

In principle, we could define an algorithm for "push" and 
"pop" primitives on the stack of v and consider a generic 
stack change as consisting of a suitable sequence of pop 
operations followed by push operations. However, this choice 
has two drawbacks: first of all, care should be taken in 
order to avoid that push and pop operations triggered by 
different network events are mixed up, resulting in inconsistent 
assigrmients of label stacks; second, implementing a stack 
change as a sequence of push and pop operations results in 
more messages being exchanged. As an example, consider 
again the network in Fig. 1 and suppose that vertex V2 has 
its stack administratively changed from S{v2) = (0 13) to 
S{v2) = (0 2 1): if this event were implemented with push 



and pop primitives, V2 would also be assigned the intermediate 
stack (0 1), which would make vi a border vertex for area 
A(Q 1 3) and cause vi to disseminate appropriate crossing (and 
final) pathlets for that area. Instead, in the final state in which 
S{v2) = (0 2 1), V\ is not supposed to disseminate these 
pathlets, because it a border vertex only for area i). We 
therefore consider the stack change as an atomic operation in 
the following. 

Stack change - We now describe the operations performed 
by a vertex when its label stack is administratively changed, 
for example because the vertex is moved to a different area. 
Despite being also modeled as a stack change, this does not 
include the case when the vertex fails, because of course it 
would not be able to undertake any actions: this case is handled 
just as if neighbors of the failed vertex received a Hello 
message from that vertex with s set to (), and is therefore 
discussed later on. 

Consider a vertex u & V and suppose its label stack S(u) 
is changed at a certain tmie instant from Sou to Snew As a 
consequence of this change, some pathlets may be created or 
deleted by u, or have their scope stack changed. The following 
steps describe how pathlets are modified by u and which 
messages are generated by u to disseminate this information. 

1) u informs all its neighbors that its label stack has 
changed. To this purpose, u sends to each of its neigh- 
bors a Hello message M with M.s = Snew^ M.d set 
according to the network destinations available at u, and 
M.a = false. 

2) u considers the atomic pathlets towards its neighbors. 
Since the stack change may influence the scope stack 
of some of these pathlets, u may have to update and 
disseminate them to a relevant subset of neighbors. 
Observe that, because of the propagation conditions and 
of the routing pohcies, an updated atomic pathlet may 
not be propagated to the same neighbors to which it 
was propagated before the stack change. Hence, u will 
send to some neighbors PatMet messages that announce 
or update some atomic pathlets, and to other neighbors 
Withdrawlet messages that withdraw atomic pathlets 
that should no longer be visible. 

Formally, for each neighbor v of u, if (SoU x Su{v)) ^ 
{Snew ^ Su{v)), u scarches !!„ for an atomic pathlet 
T^oid = {PID,u,v,aoi4,6) from u to v. Such pathlet 
must exist, because at least it has been created im- 
mediately after u has received a Hello message from 

V. Let -Knew = {FID , U, V, anew , S), with anew = 

{Snew x Su{v)) o (_L). Then, u executes procedure 

UPDATEKNOWNPATHLETSANDSTACK(w, Sold, Snew, 

{llu\{TToid}) U {TTnew}) from Algorithm 2. 

3) u considers the areas to which its neighbors belong and 
updates its role of border vertex: if u is no longer a 
border vertex for some areas after the stack change, it 
must delete all crossing and final pathlets for these areas 
and withdraw them to relevant neighbors. Conversely, if 
after the stack change u becomes a border vertex for 



some areas, it must create crossing and final pathlets 
for these areas and disseminate them to the relevant 
neighbors, according to the propagation conditions and 
to the routing policies. For the areas for which u 
continues to be a border vertex, it must check whether 
the pathlets that make up its crossing and final pathlets 
are still available, or whether new compositions are 
possible, and disseminate the corresponding information. 
To realize these operations, u executes procedure Up- 

DATECOMPOSEDPATHLETSANDSTACK(u, Sold, Snew, 

(n„\{7ro/d}) U {i^new}), which is invoked within Up- 
dateKnownPathletsAndStack. 

Update of network destinations - When an administrative 

configuration change modifies the set of network destinations 
available at a certain vertex u, all vertices that store a pathlet 
with u as an end vertex must have this pathlet updated with 
the new available destinations. To achieve this, u peforms 
only step 1) of the stack change: this is enough to propa- 
gate the updated information. In fact, as shown in the next 
subsection, when a vertex v receives a Hello or a Pathlet 
message that carries already known information but for the 
set of destinations, v updates the pathlets it stores locally and 
forwards the updated information to its neighbors, according 
to the propagation conditions and to the routing policies. 

E. Message Handling 

In the previous subsections we have described the actions 
performed by a vertex when it detects a topological change or 
it undergoes a configuration change. Therefore, to complete 
the specification of the control plane, we need to specify the 
behavior of a vertex when it receives any of the messages 
introduced in this section. Assume that vertex u receives a 
message M. The actions performed by u depend on the type 
of message M, and are detailed in the following. 

Receipt of a Hello Message - When a vertex u receives a 
Hello message M from a neighbor M.o, it performs several 
actions. 

First of all, u updates its knowledge about vertex M.o by 
setting Su{M.o) = M.s and D„(M.o) = M.d. 

After that, u checks whether M.a = True, which means 
that this is the first Hello message sent by M.o since its 
activation. If this is the case, vertex M.o needs to learn about 
all the currently available pathlets. For this reason, u sends to 
M.o all the information it currently knows, and in particular: 
for every pathlet tt = {FID,v,w,a,6) in any of the sets 
n„, Cu, and Fu kept by u such that M.o € N{u, S{u),a), 
u sends to M.o a Pathlet message Mp with Mp.p = tt, 
Mp.o = V, and Mp.t = t, where t is taken from entry 
{FID,v,a,t,+) in history (note that such an entry must 
exist for every pathlet learned or created by u). Moreover, for 
every entry {FID,v,a,t, — ) in history i7„ such that M.o G 
N{u, S{u), a), u sends to M.o a Withdrawlet message Mw 
with Mw-i = FID, Mw-s = a, Mw-o = v, and Mw-t = t. 
Observe that, in sending these messages, u preserves the origin 
vertex and timestamp of the originally leamed information, as 
specified in the history. 



At this point, u creates or updates pathlets as required, based 
on the newly learned information about its neighbor M.o. As a 
first step, u creates an atomic pathlet towards M.o, or updates 
it if it already exists in n„. Since this action may change the 
contents of n„, several crossing and final pathlets may also 
need to be created or deleted, based on the availabiUty of their 
component pathlets. Moreover, after u has learned about the 
label stack of M.o, it can detect that its status of border vertex 
for some areas has changed (it may become border vertex for 
some new areas and cease being border vertex for other areas), 
and this also requires updating crossing and final pathlets. 

More formally, let i^new = {FIDnew,u, v, CT„e™, (5„eto) be a 
new atomic pathlet from u to M.o, with FIDnew chosen to 
be unique at u, v = M.o, <Jnew = {S{u) M Su{v)) o (_L), and 
6new = Dy,{v). If a pathlet ttom = {FID oid,u,v,aoid,5ou) 
exists in n„, then let FIDnew = FIDoU (that is, the old path- 
let is updated) and Hold = {t^om}', otherwise, let Hou = 0- 
To realize all the required pathlet update operations, including 
those of crossing and final pathlets, and disseminate the 
updated information, u executes procedure UpdateKnown- 
Pathlets(u, {Uu\Uoid) U {iTnew}) from Algorithm 2. 

Note that, even if vertex M.o has sent an updated label 
stack, for example due to a stack change, it may be the case 
that no pathlets are updated by u and no messages are sent by 
u. In fact, if the atomic pathlet ttqm from m to M.o already 
existed in n„ and its scope stack (Joid and set of destinations 
Sold are unchanged in TTnew (which, for the scope stack, only 
requires that S{u) xi Su{v) is unchanged), u does not perform 
any actions: this is visible in Algorithm 2 because the two for 
cycles at lines 2 and 13 execute no iterations since n„ew = 
n„; moreover, if S{u) xi Su{v) is unchanged, u cannot change 
either the areas for which it is a border vertex or any of the 
sets Bu, and this causes sets Coid and Cnew in Algorithm 4 to 
be empty, resulting in no actions being performed even during 
the execution of that algorithm. 

Last, u checks whether the set of available destinations at 
its neighbor M.o has changed. In particular, for each pathlet 

T^oid — {FID, V, w, (J, Sold) in n„ or in any of the sets F^ such 
that w = M.o and Sold ¥^ D^iw), u sends to all its neighbors 
in N{u, S{u),a) a Pathlet message M with M.p = iVnew, 
where nnew = {FID,v,w,a, Du{w)). 

Receipt of a Pathlet Message - Upon receiving a Pathlet 
message carrying a pathlet nmsg = M.p, a vertex u first of 
all checks the freshness of the information contained in that 
message: if the information contained in the message is older 
than the information that u currently has about iTmsg, u must 
send back a message with the updated information; otherwise, 
u accepts the fresher information and updates its pathlets and 
history accordingly. 

In particular, let TTmsg = {FID,v,w,amsg,Smsg)- If u is 
the originator of nmsg, namely ii = v, then the information 
known by u about nmsg is to be considered always fresher, 
and the message can never carry updated information. If u 
is not the originator of nmsg, the freshness of message M is 
determined by comparing the message timestamp M.t with 



the timestamp of the most recent information that u keeps 
about TTmsg in its history In all the cases in which the 
information received in message M is outdated, u rephes 
with a message containing the updated information. Function 
IsPathletMessageFresher(u, M) in Algorithm 5 realizes 
this freshness check and returns True only when message M 
carries updated information. This function also sends updated 
information back to M.src, forwards the received message 
to relevant neighbors, and updates the history Hu as required. 

u therefore executes function IsPathletMessage- 
Fresher(u, M): if it returns False, the handhng of M by 
u is finished, because the message does not carry any use- 
ful information (and function IsPathletMessageFresher 
already takes care of forwarding the Pathlet message as 
appropriate). Otherwise, u looks in its set n„ for a pathlet 
TToW = {FID, v,w, a old, Sold)- If this pathlet exists, u sets 
Ilofd = {t^ou}, otherwise u sets Hou = 0- At this point, 
u updates its sets of known pathlets, crossing pathlets, and 
final pathlets, as well as its status of border vertex and 
sets i?„ of other border vertices, and disseminates updated 
information to its neighbors. All these tasks are accomplished 
by u by executing procedure UpdateKnownPathlets(u, 

(n„\no(d) U {TT^sg})- 

Receipt of a Withdrawlet Message - Handhng of a 
Withdrawlet message M received by a vertex u is much 
similar to that of a Pathlet message. First of all, u checks 
the freshness of the information carried by M by invoking 
function IsWithdrawletMessageFresher(u, M) in Al- 
gorithm 6: if this function returns False, then handhng of 
message M is completed. 

Otherwise, u searches n„ for pathlet tToU = 
{M.f, Af.o, w, M.s, 5). If this pathlet exists in n„, u updates 
pathlets and disseminates information by executing procedure 
UpdateKnownPathlets(u, Ilu\{noid}y, otherwise u 
undertakes no further actions, because there is no pathlet to 
be withdrawn. Note that, regardless of whether ttou exists in 
n„, function IsWithdrawletMessageFresher aheady 
takes care of appropriately forwarding the Withdrawlet 
message. 

Receipt of a Withdraw Message - Receiving a With- 
draw message M has the same effect of receiving several 
Withdrawlet messages, all with the same timestamp M.t, 
one for each FID of the pathlets in n„ that have scope 
stack M.s and start vertex M.o. In order to handle this type 
of message, function IsWithdrawletMessageFresher 
needs to be slightly modified as follows: if M carries fresher 
information for all the pathlets in n„ with scope stack M.s 
and start vertex M.o, then history is appropriately up- 
dated for all these pathlets and only the single Withdraw 
message is further propagated by u; otherwise, if u has a 
more recent history entry in Hu for at least one of these 
pathlets, u treats the Withdraw message exactly as a sequence 
of Withdrawlet messages, sending back to M.src single 
Pathlet and Withdrawlet messages with updated information, 
and forwarding single Withdrawlet messages as appropriate. 



If M is determined to carry fresh information, pathlets are 
then updated by u as already explained for the Withdrawlet 
message. 

VI. Applicability Considerations 

In this section we describe how the control plane we have 
formally defined can be implemented in real world, and we 
explain how further requirements, hke support for Quality of 
Service levels, can easily be accoimnodated in our model. It 
is out of the scope of this paper to detail the configuration 
language that would have to be used to configure our control 
plane. 

Technologies - The control plane we have defined in the 
previous sections is completely independent of the data plane 
that carries its messages: network destinations carried by 
Pathlet messages are completely generic and each vertex only 
communicates with its immediate neighbors, and to achieve 
this a simple link-layer connectivity is required. However, the 
information collected by our control plane can only be fully 
exploited if a data plane that can handle pathlets is available. 
As also explained in Section IV, data packets should have 
an additional header that specifies the sequence of FIDs of 
the pathlets that the packet is to be forwarded along. When a 
router receives a packet, it looks at the topmost FID, retrieves 
the next-hop router that corresponds to that FID, removes the 
FID from the packet's header, and forwards the packet to the 
next-hop router. If the FID corresponds to a crossing or final 
pathlet, the router also alters the packet's header by prepending 
the FIDs of the component pathlets before forwarding it. We 
highlight that the sequence of FIDs can be represented by a 
stack of labels and the operations we have described actually 
correspond to a label swap. For this reason, it is easy to 
implement the data plane of pathlet routing, as well as our 
control plane, on top of MPLS. The authors of [1] share the 
same vision in [15], yet they underline that MPLS does not 
allow to implement an overlay topology, which is very useful 
to specify, e.g., local transit policies. We argue that, unlike [1], 
our control plane is conceived for internal routing in an ISP's 
network, a different scenario where MPLS is a conmionly 
adopted technology and different requirements exist in terms 
of routing policies. 

Incremental Deployment - It is of course unrealistic for an 
Intenet Service Provider to change the internal routing protocol 
in the whole network in a single step. Our control plane is 
therefore designed to support an incremental deployment, so 
that a pathlet-enabled zone of the network that adopts our 
control plane and an MPLS data plane can nicely coexist with 
other non-pathlet-enabled zones of the same network that use 
different control and data planes. Assuming that non-pathlet- 
enabled zones use IP (possibly in combination with MPLS), 
we have two interesting situations: a pathlet-enabled zone is 
embedded in an IP-only network (initial deployment phase) 
or an IP-only zone is embedded in a pathlet-enabled network 
(legacy zones that may remain after the deployment). The first 
scenario can be easily implemented by making routers at the 
boundary of the two zones redistribute IP prefixes from the 



Algorithm 5 Algorithm to determine whether a PatMet message M carries updated information about a pathlet: the function 
returns True only in this case. It also handles message forwarding and history update. 
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function IsPathletMessageFresher(«, M) 

■Kmsg M.p 

Let TTmsg = {FID , V,W, (Tmsg , Smsg) 

it u = V then 

u is the originator of pathlet iVmsg 

if there is no pathlet identified by FID and with start vertex u in n„ or in any of the sets Cu and Fu then 
Mw new Witlidrawlet message 

Mw-f ^ FID 

Miv-S <— a^sg 

Send Mw to neighbor M.src 

else 

TTciir ^ pathlet identified by FID and with start vertex u that is known at u 
if Vmsg / T^cur then 

Mp new Pathlet message 

Mp.p ■(- TV cur 

Send Mp to neighbor M.src 
end if 
end if 

return False 
else 

if 3 {FID, v,a,t, type) in then 
if t < M.t then 

Replace {FID , f, (Tmsg ■) type) in Hu with (^FID, ^msgi M.t,+> 

for each n € N{u,S{u),amsg)\{M.src} do 

Send M to neighbor n 
end for 
return True 
else 

if type = + then 

TTcur pathlet in n„ identified by FID and with start vertex v 
Mp new Pathlet message 

Mp.p <— TTcur 
Mp.t t 

Send Mp to neighbor M.src 

else 

Mw new Withdrawlet message 
Mw-f ^ FID 
Mw-s a 
Mw-t t 

Send Mw to neighbor M.src 
end if 

return False 
end if 
else 

Add {FID,v,amsg,M.t,+} to 

for each n G N{u, S{u),amsg)\{M.src} do 

Send M to neighbor n 
end for 
return True 
end if 
end if 
end function 



Algorithm 6 Algorithm to determine whether a Withdrawlet message M carries updated information about a pathlet: the 
function returns True only in this case. It also handles message forwarding and history update. 

1: function IsWithdrawletMessageFresher(m, M) 

2: if 3 (M.f , M.o, (J, t, type) in Hu then 

3: lit < MX then 

4: Replace {M.t, M.o, a, t, type) in Hu with (M.f, M.o, M.s, M.t, -) 

5: for each n e N{u, S'(u), M.s)\{M.src} do 

6: Send M to neighbor n 

7: end for 

8: return True 

9: else 

10: if type = + then 

11: Ttcur <— pathlet in n„ identified by M.f and with start vertex M.o 

12: Mp ■(— new Pathlet message 

13: Mp.p TTcur 

14: Mp.t ^ t 

15: Send Mp to neighbor M.src 

16: else 

17: Mw new Withdrawlet message 

18: Mw-f M.f 

19: Mw-s M.s 

20: Miv.t t 

21: Send Mvk to neighbor M.src 

22: end if 

23: return False 

24: end if 

25: else 

26: There is no history entry for the pathlet withdrawn by M, therefore u cannot know anything about that pathlet. However, the 

Withdrawlet must still be forwarded 

27: Add (M.f, M.o, M.s, M.t,-) to Hu 

28: for each n € N{u, S{u), M.s)\{M.src} do 

29: Send M to neighbor n 

30: end for 

31: return False 

32: end if 
33: end function 



IP control plane to the pathlet control plane: this means that 
boundary routers appear as the originators of these destination 
prefixes in the pathlet zone. Each boundary router then creates 
final pathlets to get to the destinations originated by the other 
boundary routers: the IP prefixes that boundary routers learn 
from the pathlet zone in this way are then redistributed from 
the pathlet control plane to the IP control plane. Likewise, 
the IP prefixes of destinations that are available at routers 
within the pathlet zone are also redistributed to the IP control 
plane. In this way, IP-only routers can reach destinations 
inside the pathlet zone or just traverse it as if it were a 
network link. As a small exception to what we have shown in 
Section III, in this scenario boundary routers need to compose 
final pathlets even if they just belong to area A^i^)- From 
the point of view of the data plane, packets that enter the 
pathlet-enabled zone will have suitable FIDs pushed on their 
header, indicating the pathlets to be used to reach another 
boundary router or a destination within the pathlet zone; these 
FIDs will be removed when packets exit the pathlet-enabled 
zone. The second scenario can be implemented by assuming 
that routers at the boundary of the two zones have a way to 
exchange the pathlets they have leamed by exploiting the IP- 
only control plane: for example, this could be achieved by 



tunneUng Pathlet messages in IP or by transferring pathlet 
information suitably encoded in BGP messages (possibly in 
the AS path attribute). Boundary routers then redistribute from 
the pathlet control plane to the IP control plane the IP prefixes 
they have leamed fi-om the final pathlets: in this way, the 
boundary routers appear as the originators of these prefixes 
in the IP-only zone. Moreover, each boundary router will 
disseminate final and crossing pathlets that lead, respectively, 
to destinations inside the IP-only zone or to other boundary 
routers. In this way, the IP-only zone can be traversed (or its 
intemal destinations be reached) without revealing its internal 
routing mechanism, and appears just as if it were an area of the 
pathlet zone. From the point of view of the data plane, a packet 
containing FIDs in its header must be enabled to traverse 
the IP-only zone: this can be easily achieved by establishing 
tunnels between pairs of boundary routers. Of course there 
is no sharp frontier between the first and the second scenario, 
because the roles of "embedded" and "embedder" zone can be 
easily swapped: although they best fit specific phases of the 
deployment, both choices can indeed be permanentiy adopted, 
and it is up to the administrator to decide which one is it best 
to apply. 

Quality of Service - We have designed our control plane to 



support the computation of multiple paths between the same 
pair of routers. Besides improving robustness, this feature can 
also be exploited to support Quality of Service. In particular, 
each pathlet could be labeled with performance indicators 
(delay, packet loss, jitter, etc.) that characterize the quality 
of the path that it exploits. Upon creating a crossing or 
final pathlet, a router will update the performance indicators 
according to those of the component pathlets. When multiple 
pathlets are available between the same pair of routers, a 
router will be able to choose the one that best fits the QoS 
requirements for a specific traffic flow. 

Software Defined Networking - A relatively recent trend 
in computer networks is represented by the separation of the 
logic of operation of the control plane of a device from the 
(hardware or software) components that take care of actual 
traffic forwarding. This trend, known as Software Defined Net- 
working, has a concrete realization in the OpenFlow protocol 
specification [16]. We believe that our approach has several 
elements that make it compatible with an OpenFlow scenario. 
First of all, the fact that packets are forwarded according to 
the sequence of FTDs contained in their header is a form of 
source routing: this matches with the OpenFlow mechanism of 
setting up flow table entries to route all the packets of a flow 
along an estabhshed path. Moreover, a recent contribution [17J 
proposes a hierarchical architecture for an OpenFlow network: 
the authors suggest that a set of devices under the coordination 
of a single controller can be seen as a single logical device 
that is part of a larger OpenFlow network, in turn having 
its own controller. Following this approach, we could assign 
an OpenFlow controller instance to each area defined in our 
control plane, and these instances could be organized in a 
hierarchy that simply reflects the hierarchy of areas: in this 
way, each instance can direct traffic along the desired sequence 
of pathlets within the area that it controls, whereas instances 
at higher levels of the hierarchy can only see lower controllers 
as a single entity, reflecting the idea of crossing pathlet. 

VII. Experimental Evaluation 

In order to verify the effectiveness of our approach and 
to assess its scalability, we have performed several exper- 
iments in a simulated scenario. For this purpose we used 
OMNeT-H- [18], a component-based C-n- simulation frame- 
work based on a discrete event model. We considered a few 
other alternative platforms, including, e.g., the Chck modular 
router [19], but in the end we selected OMNcTh-h- because 
it has a very accurate model of a router's components, like 
Click, and it also allows to run on a single machine a complete 
simulated network with realistic parameters, such as link 
delay. Moreover, there exist lots of ready-to-use extensions 
for OMNeT-H- that allow the simulation of specific scenarios, 
including IP-based networks. To consider a reahstic setup, we 
therefore chose to build a prototype implementation of our 
control plane based on the IP implementation made available 
in the INET framework [20], a companion project of OM- 
NeT-H-. In our prototype, the messages of our control plane 
are exchanged encapsulated in IP packets with a dedicated 



protocol number in the IP header. We implemented most of 
the mechanisms described in Sections IV and V, with very 
few exceptions that are not relevant for the purposes of our 
experiments. In particular, we implemented all the message 
types (except fields carrying network destinations), all the 
propagation conditions, the mechanisms to discover border 
vertices for an area and to compose atomic and crossing 
pathlets, the history at each vertex, and a relevant portion 
of the forwarding state (the mapping between a pathlet and 
its component pathlets). Some of the algorithms adopted in 
our implementation may still not be tuned for best efficiency, 
but this is completely irrelevant because we measured routing 
convergence times by using the built-in OMNeT-i-H- timer, 
which reflects the event timings of the simulation, instead of 
the waU clock. 

Each simulation we ran had two inputs: a topology specifi- 
cation, consisting of routers, links, assignment of label stacks 
to routers, and link delays; and an IP routing specification, 
consisting of assignments of IP addresses to routers' interfaces 
and of insertion of static routes for the networks that were 
directly connected to each router. In order to facilitate the 
automated generation of large topologies, we assigned to each 
link a /30 subnet selected according to a deterministic but 
completely arbitrary pattern. 

We first executed several experiments in a small topology 
with a weU-defined structure encompassing border routers for 
several areas. This topology consisted of 15 routers, 20 edges, 
4 areas with a maximum length of the label stacks equal to 3 
(including label Iq), and at least 3 vertices in each area. This 
helped us to thoroughly verify the implementation for consis- 
tency. We then implemented a topology generator and used it 
to create larger topologies that could aUow us to assess the 
scalabiUty of our control plane. The topology generator works 
by creating a hierarchy of areas and by adding routers and 
links randomly to the areas. It takes the foUowing parameters 
as input: length N of the label stack of all the routers; number 
of routers having a stack of length N, specified as a range 
[i?min,-Rmax], with possibly i?min = ^max; uumbcr of arcas 
contained in each area, specified as a range [Amin, ^max], 
with possibly Amm = ^maxi probability P of adding an edge 
between two vertices; fraction B of the routers within an area 
that act as border routers for that area (namely that can have 
links to vertices outside that area). The topology generator 
proceeds by recursively creating areas, starting from a single 
area that comprises all vertices. The complete procedure is 
described in Algorithm 7. 

A detailed description of the experiments we carried on 
follows. This description is still in a drafty form and will be 
improved in a future release of this technical report. 

Two preliminary experiments were carried out to assess 
the scalabiUty of our control plane. In both experiments, we 
ran several simulations where the size of the topology were 
increased. To increase the size of the topology we adopted 
the following strategy. Every input parameter of the topology 
generator was fixed, except one that has been used to change 
the size of the topology. In the first experiment, the number 



of areas contained in each area was chosen as the variable 
input, while in the second experiment, the length of the label 
stack was chosen as the variable input. For each simulation, we 
collected data regarding the number of messages sent by each 
router, the number of pathlets stored in each router, and the 
convergence time of the protocol. Because of difficulty with 
the OMNeT framework, our simulation were performed with 
topologies with a Umited level of multipath. As a consquence, 
we ran simulations on topologies whose bottom-level areas 
exposes a limited level of multipath. On average, there are 
3 — 4 different paths between two border vertices of a bottom- 
level area. 

Statistical tools - The analysis of the collected data involves 
the knowledge of several widely adopted statistical tool. We 
introduce them and we try to give intuitions of their meaning. 
We use linear regression analysis to determine the level of 
correlation (linear dependence) between two arbitrary vari- 
ables X and Y. A linear regression analysis returns a line 
Iy{X) = a + BX that is an estimate of the value Y with 
respect to X. The estimation of ly depends on the specific 
measure of accuracy that is adopted. In our analysis, we 
use the Ordinary-Least-Square (OLS) method for estimating 
coefficients A and B of ly- Observe that B is extremely 
relevant in the analysis of the result since it can be interpreted 
as an estimation of the increase of Y for each additioanl unit 
of X. To verify if there exists a linear relation between X and 
Y, we look at the coefficient of determination R"^. An close 
to 1.0 indicates that the relation between X and Y is roughly 
linear, while an i?^ closer indicates that the relation does not 
seem to be linear. To determine the level of dispersion of the 
observed values for X and Y with respect to ly, we look at 
the standard error of the regression (SER) of ly. This value 
has the same unit of values in Y and can be interpreted as 
follows: approximately 95% of the points lie within 2 x SER 
of the regression Une. Sometimes, it is better to look at a 
normaUzed value that represents the level of dispersion of a set 
of values X. This can be done, by computing the coefficient of 
variation c„ of X which is independent of the unit in which 
the measurement has been taken and can be interpreted as 
follows: given a set of values X, approximately 68% of the 
values lie between — c„) and + c„), approximately 
95% of the values lie between — 2cy) and + 2c„), and 
approximately 99.7% of the values lie between — 3c„) and 
+ 3c„), where ji is the arithmetic mean of X. 

Experiment 1 - We constructed network topologies using 
the following fixed parameters: i^min = ^^max = 10, = 2, 
P = 0.1 and B = 5. The variable input ^min = ^max varied 
from 2 to 7. Roughly speaking, we keep a constant number 
of levels in the area hierarchy while increasing the number of 
areas in the same level. For each combination of these values, 
we generated 10 different topologies and for each of these 
topologies, we ran an OMNeT++ simulation and collected 
relavant data as previously specified. In Fig. 5 (Fig.4) we 
show that the maximum (average) number of pathlets stored 
in each router (depicted as crosses) grows with respect to the 
number of edges in the topology. In particular, the growth 



is approximately linear as confirmed by a linear regression 
analysis (depicted as a line) performed on the collected data. 
The slope of the line is 0.83 (0.75), which can be interpreted 
as follows: for each new edge in the network, we expect that 
the value of the maximum (average) number of pathlets stored 
in each router increases by a factor of 0.83 (0.75) on average. 
To assess the linear dependence of the relation, we verified 
that the i?^ = 0.87 (i?^ = 0.9), which means that the linear 
regression is a good-fit of the points. The standard error of 
the regression is 39.8 (9.26), which can be interpreted as 
follows: approximately 95% of the points he within 2 x 39.8 
(2 X 9.26) of the regression line. We motivate the linear growth 
as follows. Observe that, each topology has a fixed number 
^max of areas and therefore the expected number of crossing 
pathlets created for any arbitrary area is the same. Hence, by 
increasing Amax, since the number of crossing pathlet created 
in each area grows Unearly with ^max. we have that also the 
number of pathlets stored in each router, which contains a 
constant number of atomic pathlets plus each crossing pathlet 
created by border router of neighbors areas, grows Unearly. In 
Fig. 3 (Fig. 2) we show that similar results hold also when we 
analyze the average/maximum number of messages by each 
router. In fact, we observed that many considerations that we 
observed with respect to the number of messages sent by each 
router, are also valid with respect to the number of pathlets 
stored in each router. In fact, as shown in Fig. 7, we show that 
there exists an interesting linear dependence, with R'^ = 0.87, 
between the number of messages sent by each router and the 
number of pathlets stored in each router. 

Experiment 2 - We constructed network topologies using 
the following fixed values: i?min = ^max = 10, A„iin = 
^max = 2, P = 0.1 and B = 5. The variable input N 
varied in the range between 1 and 4. Roughly speaking, we 
keep a constant number of subareas inside an area, while we 
increase the level of area hierarchy. For each combination of 
these values, we generated 10 different topologies and for each 
of these topologies, we run an OMNeT++ simulation and the 
same data as in the first experiment. In Fig. 1 1 (Fig. 10) we see 
that the maximum (average) number of pathlets stored in each 
router (depicted as crosses) grows linearly with respect to the 
number of edges in the topology. A hnear regression analysis 
computed over the collected data shows that the slope of the 
line is 1.38 (0.94), which can be interpreted as follows: for 
each new edge in the network, we expect that the value of the 
maximum (average) number of pathlets stored in each router 
increases by a factor of 1.38 (0.94) on average. To assess 
the hnear dependence of the relation, we verified that the 
i?2 = 0.75 (i?^ = 0.81), which can still be considered a good- 
fit of the points. As for the dispersion of the points with respect 
to the line, it is easy to observe that, the higher the number 
of edge, the higher the dispersion. If we look to the standard 
error of the regression, since the measure is not normalized, we 
may obtain an inaccurate value of the dispersion. Therefore, 
we do the following. We consider 4 partition Pi , . . . , P4 of the 
measurents, where Xi contains each measure collected when 
N = i, and compute the coefficient of variation Cy of each 



subset. We observe that Cy varies between 0.21 and 0.42 (0.24 
and 0.31), which means that in each partition Xi, 95% of the 
points lie within 2 • 0.41/i (2 • 0.31/i) of the mean /.i of Xi. We 
have not enough data to check whether there exists a lineaer 
dependence between the value c„ of a partition Xi and the 
index i. 

We motivate the linear growth as follows. Observe that 
each bottom area construct on average the same number 
of crossing pathlets, regardless the levels of the hierarchy. 
Because of the low values chosen for P, Amin, and Amax, 
the number of crossing pathlets created by higher areas is 
roughly proportional to the number of crossing pathlets created 
by its subareas. Now, consider the set of pathlets stored in a 
router with stack label (xi . . . a;„). It contains atomic pathlets 
for the areas in which it belongs, which are logarithmic with 
the number of edges, and it contains crossing pathlets from 
other areas, which are roughly proportional to the number of 
bottom-level areas. Hence, each time N is increased, both the 
number of edges and the number of bottom-level areas, grows 
exponentially with the same rate, and therefore the number of 
pathlets stored in each router increases linearly with respect 
to the number of edges. As for the first experiment, a similar 
trend can be seen also in Fig. 9 and Fig. 8 with respected to the 
maximum and average number of messages sent by a router, 
respectively. In fact, also in this experiment, we observe that 
there is a good linear relation between the number of messages 
sent by a vertex and the number of pathlets stored in each 
router (see Fig. 13). 

Convergence time - We now consider the convergence 
time T of the protocol expressed in milliseconds. In both 
experiments we set link delays as random uniform variable in 
the range between 10 and 50 milliseconds. Convergence time 
for the first and the second experiments with respect to the size 
of the network (expressed by the number of edges) are shown 
in Fig. 6 and 12, respectively. In Experiment 1 we achieved 
T e [363,611] whereas in Experiment 2 T e [259,736]. 
The minimum value in both the intervals correspond to the 
minimum value among the 10 different generated topologies 
with the value A — 2 and the value L = 1 respectively 
for Experiment 1 and Experiment 2; on the other hand the 
maximum value in both the intervals is the maximum value 
among the 10 different generated topologies with the value 
L = 7 and the value L — A respectively for Experiment 1 
and Experiment 2. 

In the first experiment, the value T — 363 msec was 
achieved for a topology with two areas, A(o i) and A(o 2). 
both contained into area ^q, with i?min — Rmax = 10 
vertices per area. By random adding edges, we obtained a 
network with 20 vertices and 22 edges. On the other hand, 
the value T = 611 msec was achieved for a topology with 
Rmin = Rmax = 10 vcrticcs per bottom-lcvcl area and An-iin = 
^max = 7 bottom-level areas contained into area Aq. By 
random adding edges, we obtained a topology with 70 vertices 
and 89 edges. As for the experiments that involve the lenght 
of the label stack, we obtained the value T = 0.259 msec for 
a network topology where all vertices belong only to the same 
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Fig. 2. Average number of messages sent by a router with respect to the 
number of edges contained in the topology. (Experiment 1) 

area Ao. The generated network has 10 vertices and 12 edges. 
On the other hand, the value T = 736 msec was achieved 
for a network topology with i?inin = ^max = 10 vertices per 
bottom-level area and the length of the label stack is A^ = 4. 
The generated topology has 80 vertices and 104 edges, with 
vertices organized into a network that have 2 areas at each 
level. In particular, Aq contains two areas, Aj-q i) and A(q 2). 
Each of these areas contains, in turn, two areas: A(o 1 2) ™d 
A(Q 1 3) are contained into A(^q 1), while 2 3) and yl(o 2 4) 
are contained into Ajq 2) and so on. 

In both experiments, we achieved a convergence time below 
Isec and we observed that the correlation between the size 
of the network and the convergence time, does not exhibit a 
linear behaviour. In Fig. 6 it can clearly be observed that the 
regression line does not fit well the points. The slope of the 
regression line is 1.2, which can be interpreted as follow: for 
each new edge in the network, we expect that convergence 
time increases of 1.2 milliseconds. However, we computed an 
value of 0.26, which is close to and suggest a lack of 
correlation between convergence time and number of edges. 
In other word, the convergence time is independent from the 
number of edges. On the other hand. In Fig. 12 a more strong 
correlation between convergence time and number of edges 
seems to hold. In this case, the slope is 2.9 with an value 
of 0.57. This means that the growth appears to be linear 
but there is a high dispersion of the points from the line. 
We recall that both experiments has been run with a pathlet 
composition rule that force each border vertex to compose all 
the possible pathlets to every other border vertex of the same 
area. Changing this rule, we expect that convergence times 
will decrease. 

Our prototype implementation, including the topology gen- 
erator, is publicly available at [21]. 

VIII. Conclusions and Future Work 

In this paper we introduce a control plane for internal 
routing inside an ISP's network that has several desirable 



Algorithm 7 Algorithm used in our topology generator. 

function P0PULATEAREA(^, level, -Rmin, -Rmax, N, Amin, ^max, P, B) 

if level = N then 

r ^ a random number in [iimim -Rmax] 

Add r vertices to A 

repeat 

for each pair {u, v) of vertices in A do 

Add an edge between u and v with probabiUty P 
end for 
until A is connected 

Randomly pick r x B routers in A and mark them as border routers for A 
else 

a ^ a random number in [vlmin, ^max] 

Create a areas inside A; let A be the set of these areas 

for each A in ^ do 

P0PULATEAREA(A level + 1, i?min, i?max, N, A^in, ^max, P, B) 

i? i? U all border routers for A 
end for 
repeat 

for each pair (w,, w) of routers in R do 
Flip a coin with probabiUty P 
if heads then 

Add an edge between u and v 
E ^ E[J{u,v) 
end if 
end for 

until the undirected graph formed by vertices in R and edges in E is connected 
Randomly pick \R\x B routers in R and mark them as border routers for A 
end if 
return A 
end function 

function TOPOLOGYGENERATOR(iirnin, -Rmax, N, A^i-a, Amax, P, B) 

Create an area A 
level 1 

return P0PULATEAREA(A, level, Rmin, -Rmax, N, A^in, ^max, P, B) 

end function 



properties, ranging from fine-grained control of routing paths 
to scalability, robustness, and QoS support. Besides intro- 
ducing the basic routing mechanisms, which are based on 
a well-known contribution [1], we provide a thorough and 
formally sound description of the messages and algorithms 
that are required to design such a control plane. We validate 
our approach through extensive experimentation in the OM- 
NeT++ simulator, which reveals very promising scalability and 
convergence times. Our prototype implementation is available 
at [21]. 

There are a lot of improvements that we are still interested 
in working on. Some of them are possible optimizations, 
whereas others are foundational issues that are still open: 
here we mention a few. Our current choice of messages types 
imposes a strong coupling between routing paths and network 
destinations: if a network destination changes its visibiUty (for 



example, a router starts announcing a new IP prefix), several 
pathlets to that destination must be (re)announced, even though 
the routing has not changed. Inspired by recent research 
trends [22], we could change the protocol a bit to separate 
these two pieces of information. In line with this decoupling 
requirement, we would like to investigate on how to deal 
with dynamic changes in QoS levels associated with pathlets. 
Routing policies, especially pathlet composition rules, could 
of course be refined to accommodate further requirements 
that we have not considered yet. Moreover, their specification 
and application could be enhanced to improve scalability in 
common usage scenarios (for example, when several areas are 
grouped into a larger one). The pathlet expiration mechanism 
needs further improvements to correctly purge pathlets in the 
presence of routing policies. The handling of stack change 
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Fig. 3. Maximum number of messages sent by a router with respect to the Fig. 6. Network convergence time with respect to the number of edges 
number of edges contained in the topology. (Experiment 1) contained in the topology. (Experiment 1) 
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Fig. 4. Average number of pathlets stored in a router with respect to the 
number of edges contained in the topology. (Experiment 1) 



Fig. 7. Maximum number of pathlets stored in a router with respect to the 
maximum number of messages sent by a single router. (Experiment 1) 
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Fig. 5. Maximum number of pathlets stored in a router with respect to the 
number of edges contained in the topology. (Experiment 1) 



Fig. 8. Average number of messages sent by a router with respect to the 
number of edges contained in the topology. (Experiment 2) 
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Fig. 9. Maximum number of messages sent by a router witli respect to tlie 
number of edges contained in tlie topology. (Experiment 2) 
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Fig. 10. Average number of pathlets stored in each router with respect to 
the number of edges contained in the topology. (Experiment 2) 
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Fig. 1 1 . Maximum number of pathlets stored in a router with respect to the 
number of edges contained in the topology. (Experiment 2) 
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Fig. 12. Network convergence time with respect to the number of edges 
contained in the topology. (Experiment 2) 
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Fig. 13. Maximum number of pathlets stored in a router with respect to the 
number of messages sent by a vertex. (Experiment 2) 

events could also be improved: in particular, we could design 
more effective mechanisms to transparently replace a pathlet 
that is no longer visible with other newly appeared pathlets, 
without spreading messages to the whole network. Being 
modeled as stack change events, the handling of faults could 
be improved likewise. 
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